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ABSTRACT 

This paper is a report on the estimated effects on students 
and families of the offer of a voucher (the "intention-to-treat " effect) and 
the effects of switching from a public to a private school (the "treatment- 
on-the-treated" effect). Specifically, the report estimates the impact of 
voucher programs on student test scores, parental satisfaction with their 
children's school, and parental reports of the characteristics of the school 
their children attended. Data for the report were gathered from test scores 
and parent surveys in schools with voucher programs in Washington, D.C.; New 
York City; and Dayton, Ohio. Following are three major findings of the study: 
(1) In the three cases taken together, effects of school vouchers on average 
test performance were found only for African-American students, whose scores 
were higher than scores of African-American students in public school; (2) 
Families that used vouchers to attend private schools were much more 
satisfied with their schooling than were families who remained in public 
schools; and (3) The educational environment of private schools was more 
conducive to learning than was that of public schools. (Contains 11 tables in 
the body, 7 tables in 2 appendices, and 39 endnotes, some with references.) 
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School Vouchers: Results from Randomized Experiments 



In the past decade much has been learned about the way in which school vouchers 
impact low-income families and their children.^ Ten years ago, the empirical informadon 
available about this widely debated quesdon came primarily from a flawed public-school 
choice intervendon attempted in Alum Rock, California during the 1960s.^ But in the early 
and mid-1990s, new voucher programs sprouted across the country in such cides as 
Milwaukee, Dayton, Cleveland, Indianapolis, San Antonio, Washington, D.C. and New York 
City. InidaUy, many of the evaluadons of these innovadons were limited by the quality of 
the data or the research procedures employed. Often, planning for the evaluadon began 
after the experiment was underway, making it impossible to gather baseline data or to ensure 
the formadon of an appropriate control group. As a result, the quality of the data collected 
was not as high as researchers normally would prefer.^ 

Despite their Hmitadons, these early evaluadons provided program operators and 
evaluadon teams with opportunides to learn the problems and pitfalls accompanying the 
study of school vouchers. Subsequent evaluadons of voucher programs in New York, 
Washington, D.C., and Dayton, Ohio have been designed in such a way as to allow for the 
coUecdon of higher-quality informadon about student test-score outcomes and parental 
assessments of public and private schools. Because vouchers in these cides were awarded by 
lot, program evaluadons could be designed as randomized field trials. Prior to conducdng 
the lotteries, the evaluadon team collected baseline data on student test scores and family 
background characterisdcs. One, two, and three years later, the evaluadon team again tested 
the students and asked parents about their children’s school experiences."^ In the absence of 
response biases that are condidonal on treatment status, any stadsdcaUy significant 
differences between students offered a voucher and those not offered a voucher may be 
attributed to experiences at school, because average student inidal abilides and family 
backgrounds are similar between the two groups. 

This chapter reports the esdmated effects on students and families of 1) the offer of 
a voucher on students and parents, often described as the Intendon-To-Treat (ITT) effects; 
and 2) switching from a public to a private school, commonly referred to as the effect of 
Treatment-On-the-Treated (TOT). Students who were evaluated entered private school in 
grades 2-5 in New York City and grades 2-8 in Washington, D.C. and Dayton (and other 
parts of Montgomery County, Ohio).^ Specifically, the evaluadon esdmates the impact of 
the program on student test scores, parent sadsfacdon with their child's school, as well as 
parent reports of the characterisdcs of the schools the child attended. 

The Three Voucher Programs 

The design of the three voucher programs was similar in key respects, thereby 
allowing the evaluadon team to combine results from the separate evaluadons of these 
programs. All were privately funded; aU were targeted at students from low-income families, 
most of whom lived within the central city; aU provided pardal vouchers which the family 
was expected to supplement from other resources. AH students included in the evaluadon 
had previously been attending public schools. The programs, however, did differ in size, 
dming and certain administradve details. In this secdon we describe the main characterisdcs 
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of the School Choice Scholarships Foundation (SCSF) program in New York City, the 
Washington Scholarship Fund (WSF) program in Washington, D.C., and the Parents 
Advancing Choice in Education (PACE) program in the Dayton metropolitan area. 

SCSF Program in New York City 

In February 1997 SCSF announced that it would provide 1,300 scholarships worth 
up to $1,400 annually for at least three years to children from low -income families currently 
attending public schools. The scholarship could be applied toward the cost of attending a 
private school, either religious or secular. After announcing the program, SCSF received 
initial applications from over twenty thousand students between February and late April 
1997. 



To be eligible for a scholarship, children had to be entering grades one through five, 
live in New York City, attend a public school at the time of application, and come from 
families with incomes low enough to qualify for the U. S. government’s free school lunch 
program. To ascertain eligibility, students and an adult member of their family were asked to 
attend verification sessions during which family income and the child's public-school 
attendance were documented. 

Subsequent to the lottery, SCSF assisted families in identifying possible private 
schools their children might attend. By the end of the second year, about 82 percent of the 
students participating in the evaluation were using a scholarship; 79 percent of the 
participating students used the voucher for two full years, and 70 percent for three full 
years.^ 

PACE Program in Dayton, Ohio 

In the spring of 1998, Parents Advancing Choice in Education (PACE), a privately 
funded non-profit corporation, offered low-income families within the Dayton metropolitan 
area an opportunity to win a scholarship to help defray the costs of attending the school of 
their choice. Eligible applicants participated in a lottery in which winners were offered a 
scholarship that could be used at participating private and public schools in Dayton and in 
other parts of Montgomery County, Ohio. Students entering kindergarten through twelfth 
grade qualified. For the 1998-99 school year, PACE offered scholarships to 515 students 
who were in public schools and 250 students who were already enrolled in private schools. 

The program was announced in January 1998. Based on census data and 
administrative records, program operators estimated that approximately 32,000 students met 
the program’s income and eligibility requirements. Interested families were asked to call 
PACE, which took preliminary applications from over 3,000 students. PACE asked 
applicants to attend sessions where administrators verified their eligibility for a scholarship, 
students took the Iowa Test of Basic Skills (ITBS), and parents completed questionnaires. 
Over 1,500 applicants attended these verification sessions in February, March and April 
1998. The lottery was then held on April 29, 1998. 

During the first year of the program, the PACE scholarships covered 50 percent of 
tuition at a private school up to a maximum award of $1,200. Support was guaranteed for 
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eligible students for at least four years; in addition, the program expects to support students 
through the completion of high school, provided funds remain available. Scholarship 
amounts were increased beginning in 1999 as a result of increased funds available to PACE 
and support for the program by the Children's Scholarship Fund, a nationwide school-choice 
scholarship program. 

Among the public school students offered a scholarship, 78 percent of the students 
pardcipating in the evaluadon used the scholarship to attend a private school in the 
program’s first year and 60 percent were using the scholarship after two years. 

WSF Program in Washington, D.C. 

The Washington Scholarship Fund (WSF), a privately funded school voucher 
program, was originally established in 1993. At that dme, a limited number of scholarships, 
which could be used at a private school of the family's choice, were offered to students from 
low-income families. By the fall of 1997, WSF was serving approximately 460 children at 72 
private schools. WSF then received a large infusion of new funds from two philanthropists, 
and a major expansion of the program was announced in October 1997. Both general news 
announcements and paid adverdsing were used to publicize the enlarged school-choice 
scholarship program. WSF announced that, in the event that applicadons exceeded 
scholarship resources, winners would be chosen by lottery. The program expanded further 
in 1999 with support from the Children's Scholarship Fund. 

To qualify, applicants had to reside in Washington, D.C. and be entering grades K-8 
in the fall of 1998. WSF awarded parents with incomes at or below the poverty line vouchers 
that equaled 60 percent of tuidon or $1,700, whichever was less. Families with incomes 
above the poverty line received smaller scholarships. The maximum amount of tuidon 
support for high school students was $2,200. WSF has said that it will attempt to condnue 
tuidon support to the children in its program for at least three years and, if funds are 
available, undl they complete high school. No family with income above 2.7 dmes the 
poverty line was eligible for support. 

Over 7,500 telephone applicadons to the program were received between October 
1997 and March 1998; in response to invitadons sent by WSF, over 3,000 applicants 
attended verificadon and tesdng sessions. The lottery selecdng scholarship winners was held 
on April 29, 1998. WSF awarded over 1,000 new scholarships that year, with 811 going to 
students not previously in a private school. 

Provided they gained admission, scholarship students could attend any private school 
in the Washington area. During the 1998-99 school year, students pardcipadng in the 
evaluadon attended seventy-two different private schools. Of those students offered 
scholarships who pardcipated in the evaluadon, 68 percent made use of them to attend a 
private school in the first year of the program. Take-up rates declined to 47 percent in the 
second year, and to just 29 percent at the end of the third year. 

Evaluadon Procedures 
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The evaluation procedures used in all three evaluations conform to those used in 
randomized field trials. The evaluation team collected baseline data prior to the lottery, 
administered the lottery, and then collected follow-up information one and two years later. 
The following secdon details the steps taken to collect the relevant informadon. 

Baseline Data Collecdon 

During the eligibility verificadon sessions attended by voucher applicants, students in 
first grade and higher took the Iowa Test of Basic Skills (ITBS) in reading and mathematics. 
The sessions took place during the months of February, March and April immediately prior 
to the voucher lottery and generally lasted about two hours. The sessions were held in 
private-school classrooms, where schoolteachers and administrators served as proctors 
under the overall supervision of the evaluadon team and program sponsors. The producer 
of the ITBS graded the tests.’ Students in grades four through eight also completed a short 
quesdonnaire inquiring about their school experiences. 

While children were being tested, adults accompanying them filled out surveys that 
asked about their sadsfacdon with their children’s schools, their involvement in their 
children's educadon, and their demographic characterisdcs. Parents completed these 
quesdonnaires in rooms separate from those used for tesdng. Administrators explained that 
responses to the quesdonnaire would be held in strict confidence and would be used for 
stadsdcal purposes only. Respondents had considerable dme to complete their surveys, and 
administrators were available to answer quesdons about the meaning of pardcular items. 
Extensive informadon from these surveys has been reported elsewhere.^ 

Over 5,000 students pardcipated in baseline tesdng in New York City. After 
vouchers were awarded, approximately 1000 families were selected at random from those 
who did not win the lottery to comprise a control group of approximately 960 families.^ 

In Dayton, 1,440 students were tested at baseline and 1,232 parent quesdonnaires 
were completed. Of the 1,440 students, 803 were not at the dme attending a private school; 
of the 1,232 parent quesdonnaires, 690 were completed by parents of students who were not 
attending a private school. Follow-up tesdng information is reported only for students who 
were in public schools at the dme of apphcadon. 

In Washington, D.C., 2,023 students were tested at baseline; 1,928 parent surveys 
asking quesdons about each child were completed; 938 student surveys were completed. Of 
the 2,023 students tested, 1,582 were not attending a private school at the dme of apphcadon 
for a scholarship; of the 1,928 parent quesdonnaires, 1,446 were completed by parents 
whose children were not then attending a private school. Follow-up tesdng and survey 
informadon was obtained only from families with children not in private schools at the dme 
of apphcadon. 

The Lottery 

The evaluadon team conducted the lotteries in May 1997 in New York City and 
April 1998 in Dayton and D.C. Program operators nodfied lottery winners shordy 
thereafter. If a family was selected, all children in that family entering eligible grades were 
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offered a scholarship. In order to ensure that an adequate number of scholarships were 
given to students not currendy attending a private school, separate lotteries were held in 
Dayton and D.C. for students then in public and private schools. This procedure also 
assured random assignment to test and control groups of those families pardcipadng in the 
evaluation. 

In New York City, Mathematica Policy Research (MPR) administered the lottery; 
SCSF announced the winners. SCSF decided in advance to allocate 85 percent of the 
scholarships to applicants from public schools whose average test scores were less than the 
citywide median. Consequently, applicants from these schools, who represented about 70 
percent of all applicants, were assigned a higher probability of winning a scholarship. In the 
information reported in the tables, results have been adjusted by weighting cases 
differentially so that they can be generalized to all eligible applicants who would have come 
to the verification sessions had they been invited, regardless of whether or not they attended 
a low-performing school. 

Because vouchers were allocated by a lottery conducted by the evaluation team, 
those offered scholarships are not expected to differ significantly from members of the 
control group (those who did not win a scholarship). For all three cities, baseline data 
confirm this expectation. For instance, in D.C., the baseline test scores of those entering 
grades two through eight who were offered a voucher averaged 29.6 national percentile 
points in reading and 23.3 in mathematics; those not offered the scholarship scored, on 
average, 30.6 national percentile points in reading and 23.1 points in math. As in D.C., the 
demographic characteristics of those offered vouchers in Dayton and New York did not 
differ significantly from the characteristics of those who were not offered a voucher.^^ 

Collection of Follow-up Information 

The annual collection of follow-up information commenced in New York City in the 
spring of 1998 and in Dayton and D.C. in the spring of 1999. Data collection procedures 
were similar across cities. 

New York City: Testing and questionnaire administration procedures were similar to 
those that had been followed during the baseline sessions. Adult members of their family 
completed surveys that asked a wide range of questions about the educational experiences of 
their oldest child within the age range eligible for a scholarship. Students completed the 
ITBS and short questionnaires. Both the voucher students and students in the control 
group were tested in locations other than the school they were currently attending. 

SCSF conditioned the renewal of scholarships on participation in the evaluation. 
Also, non-scholarship winners selected to become members of the control group were 
compensated for their expenses and told that they could automatically reapply for a new 
lottery if they participated in these follow-up sessions. Detailed response rate information 
for the follow-up survey and testing sessions are reported in Appendix A.^' 

Washington, D.C. and Dayton: In D.C. and Dayton, the evaluation team began 
collecting follow-up information between late February and late April of 1999. As in New 
York, the procedures used to obtain follow-up data were essentially the same as those used 
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to collect baseline data. Students again took the ITBS in mathematics and reading. 
Caretakers accompanying the child completed surveys that asked a wide range of quesdons 
about the educadonal experiences of each of their children. Students in grades four through 
eight also completed a quesdonnaire that asked them about their experiences at school. 
Tesdng and quesdonnaire administradon procedures were similar to those that had been 
followed at baseline.’^ The Dayton evaluadon was concluded after two years; in D.C., 
however, a third-year follow up collecdon of tesdng and survey informadon was conducted 
in 2001. 

To obtain a high pardcipadon rate in the follow-up data collecdon effort, those who 
had declined the offer of a voucher and members of the control group were compensated 
for their expenses. They were also told in Washington, D.C. that if they pardcipated in the 
follow-up sessions, they would be included in a new lottery. In Dayton, a second lottery was 
promised as a reward for pardcipadng in the first foUow-up session. In the second year, 
however, Dayton families were only given a higher level of compensadon for pardcipadon. 

Because test-score results from the second and third-years of the evaluadon differ 
significandy between African American students and those from other ethnic backgrounds, 
the ethnic composidon of the students pardcipadng in the evaluadon is pardcularly salient. 
Forty- two per cent of the students pardcipadng in the second year of the evaluadon in New 
York City were African Americans. The percentages in Dayton and D.C. were 74 percent 
and 95 percent, respecdvely. Hispanic students pardcipadng in the second year of the 
evaluadon consdtuted 51 percent of the total in New York City, 2 percent in Dayton, and 4 
percent in Washington, D.C. Finally, 5 percent of the students pardcipadng in the 
evaluadon in New York City were white. The percentages of whites in Dayton and D.C. 
were 24 percent and 1 percent, respecdvely. The remaining students came from a variety of 
other ethnic backgrounds. 



Data Analysis and Repordng Procedures 

The evaluadon takes advantage of the fact that a lottery was used to award 
scholarships. As a result, it is possible to compare two groups of students that were similar, 
on average, except that members of the control group were not offered a scholarship. Any 
stadsdcaUy significant differences between the two groups may be attributed to the school 
experience, not the child’s inidal ability or family background, which were essendaUy the 
same at baseline. The only major threat to the validity of this causal inference would be one 
of differendal response to follow-up tesdng by members of the treatment and control 
groups based on condidons that developed after they were baseline tested. We discuss that 
possibility in the Appendix A. 

This paper provides data that help answer two quesdons. The first is: 

What was the impact on educadonal outcomes of an offer a voucher to low-income 
families residing within a large central city? This is the Intendon-To-Treat or ITT 
effect of the voucher. 
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The ITT effect compares educational outcomes of those who were offered a scholarship 
with those who were not offered a voucher. To compute program impacts on children’s test 
scores, we estimated a statistical model that took into account students’ scholarship or 
control-group status as well as baseline reading and math test scores. Baseline test scores 
were included to: 1) adjust for minor baseline differences between the treatment and control 
groups on the achievement tests; and 2) to increase the precision of the estimated impacts. 

Generalization from these results has the important disadvantage of assuming that 
usage rates of scholarships are fixed. Depending upon the size of the scholarship, the time 
the scholarship is offered, and the marketing of the program as a whole, however, usage 
rates might be highly variable. Consequendy, we report ITT results for test scores in 
Appendix B. In the text of this chapter we report answers to a second quesdon: 

What was the impact on educadonal experiences, parental sadsfacdon and test-score 
performances of students from low-income families residing within a large central 
city one, two and three years after switching from a public to a private school? This 
is the Treatment-On-the-Treated or TOT effect of the voucher. 

In medical research, the parallel quesdon is: What are the consequences of actually taking a 
pill, as prescribed? In the case of the educadon intervendon evaluated here, the answer to 
this quesdon requires a comparison between those students who were offered vouchers that 
switched from a public to a private school with public-school students who would have 
switched to a private school had they been offered a voucher. To compute the program’s 
impact on those who used a scholarship to attend a private school, we esdmated two-stage 
least squares models. The instrument is the voucher lottery, which is highly correlated with 
attendance at a private school but, because it is randomly determined, is obviously 
uncorrelated with the error term in the second-stage equadon. As a result, we are able to 
provide an unbiased esdmate of the effects of switching to a private school. 

The paper reports the TOT impact on students school experiences, parental 
sadsfacdon, and test score performance of a switch from a public to a private school for 
one, two and three years. Second and third year results compare those in private schools for 
two or three years with comparable members of the control group that were not in private 
school for two and three years, respecdvely. 

Test Score Findings 

We compare the performance of public and private school students on the Iowa 
Test of Basic Skills (ITBS) in reading and mathemadcs, as well as their combined 
performance in both subject areas. Scores range between 0 and 100 Nadonal Percendle 
Ranking (NPR) points, with the nadonal median located at the 50th percendle. The results 
reported below represent the first student-achievement informadon from randomized field 
trials on the effects of school vouchers. However, they do not so much break new ground 
as build upon a body of research that has explored the differences between schooling for 
low-income minorides in the public and private sectors. 

Prior Research 
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Several studies have compared the attainment levels and test performance of students in 
public and private schools, and they usually find that low-income and African American 
students attending private schools outperform their public-school peers. According to a recent 
analysis of 12,000 students in the National Longitudinal Survey of Youth, for instance, even 
when adjustments are made for family background, students from all racial and ethnic groups 
are more likely to go to college if they attended a Catholic school, but the effects are the 
greatest among urban minorities. This study’s findings are consistent with other studies. 

After reviewing the literature on school effects on learning. University of Wisconsin Professor 
John Witte concludes that studies "indicate a substantial private school advantage in terms of 
completing high school and enrolling in college, both very important events in predicting future 
income and well-being. Moreover, . . . the effects were most pronounced for students with 
achievement test scores in the bottom half of the distribution."^’ 



Even the most careful of these studies, however, can take into account only observed 
family background characteristics. They cannot be sure that they have taken into account an 
intangible factor — the willingness of a family to pay for their child’s tuition, and all that this 
implies about the importance they place on education. As a result, it remains unclear whether 
the findings from these studies describe actual differences between public and private schools 
or simply differences in the kinds of students and families attending them.^^ 



The best solution to the self-selection problem is the random assignment of students 
to test and control groups. Until recently, evaluations of voucher programs have not utilized 
a random-assignment research design and therefore have not overcome the possible 
selection problems. Privately funded programs in Indianapolis, San Antonio, and Milwaukee 
admitted students on a first-come, first-served basis. In the state-funded program in 
Cleveland, although scholarship winners were initially selected by means of a lottery, 
eventually all applicants were offered a scholarship, thereby precluding the conduct of a 
randomized experiment. The public Milwaukee program did award vouchers by a lottery, 
but data collection was incomplete.^^ 

As a consequence, the findings presented here on New York, D.C., and Dayton 
provide a unique opportunity to examine the effects of school vouchers on students from 
low-income families who live in central cities. In contrast to prior studies, random 
assignment was conducted by the evaluation team, follow-up test-score information was 
obtained from about one-half to two-thirds of the students who participated in the lottery, 
and baseline data provided information that allowed the analysts to adjust for non-response. 

Impacts of Private-School Attendance on Test Scores 

In interpreting the findings reported below, emphasis is placed on the estimates of 
effects on combined test scores for all three cities, taken together. The average estimates of 
the impacts of a voucher offer in all three cities provide a better indication of programmatic 
effects than do the results from any one city, because minor fluctuadons in data collection 
may influence results in any one site. Also, when student performance is estimated on the 
basis of one-hour testing sessions, combined test-score performance of students on the 
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reading and math tests is a better indicator of student achievement than either test separately. 
Theoretically, the more test items used to evaluate performance, the more likely it is that one 
will estimate performance accurately. Empirically, performances on the two tests are highly 
correlated with one another (r equals about .7). In addition, results from the two tests, when 
combined together, were found to be more stable across time and from place to place, 
indicating that combining results from the two tests reduces idiosyncratic variations in 
observations of student performance.^^ 

As can be seen in Table 1, the impact of private school attendance on student test- 
score performance differed for African Americans and members of other ethnic groups. 

One observes no significant differences between the test-score performance of non-African 
American students switching from a public to a private school and the performance of their 
peers in the control group - either after one, two or three years. Nor were significant 
differences observed in the test-score performance of these students on reading and math 
tests, considered separately. 

The effects of switching to a private school on African American students differed 
markedly from the effects on students from other ethnic backgrounds. In the three cities, 
taken together, African American students who switched from public to private schools 
scored, after one year, 3.9 NPR points higher on the combined math and reading tests, and, 
after two and three years, 6.3 percentile and 6.6 points higher, respectively, than the African 
American students in the control group. These are the average results for the three cities 
combined, weighting each city estimate in inverse proportion to its respective variance. 

The findings for each city are reported in Tables 2, 3, and 4. The largest differences 
after three years between African American students who switched from public to private 
schools and those in the control group were observed in New York City. In this city, black 
students attending private schools for three years scored 9.2 percentile points higher on the 
two tests combined than did students in the control group. In D.C., however, no significant 
differences were observed after three years, despite the fact that large two-year effects were 
observed. In Dayton, the difference in combined test-score performance was 6.5 percentile 
points after two years; the Dayton evaluation ended after two years. 

The trend over time also varies from one city to the other. As can be seen in Table 
2, in New York City, substantial test-score differences between African American students in 
private and public schools appear at the end of the first year (5.4 percentile points), attenuate 
slighdy in the second year (4.4 points), but increase to 9.2 percentile points in Year 3. In this 
city, test score gains appeared to grow over time. 

In Dayton, there was a steady upward trend in the combined test score performance 
of African Americans between year one and two. Table 3 shows that African American 
students who switched from public to private school performed 3.3 percentile points higher 
on the combined test in year one and 6.5 percentile points higher in year two. Once again, a 
model of accumulated gains could account for the findings. 

The most uncertain results for African Americans come from Washington, D.C.. As 
can be seen in Table 4, no significant differences were observed in year one, a large impact 
was observed after two years, but no impact was observed at the end of year three. Three 
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factors could account for such disparate findings. First, because only 29 percent of the 
students in the evaluation continued to use the voucher after three years (as compared to 70 
percent in New York City), third-year estimations are quite imprecise. Second, the voucher 
experiment in D.C. was contaminated by the inauguration of a charter-school initiative that 
gave families more choice than those available in New York City; indeed, 17 percent of the 
treatment group and 24 percent of the control groups attended charter schools in the third 
year of the evaluation. Finally, the differences in the third-year results might be attributed to 
the more established private sector in New York City than in Washington, D.C. CathoHc 
schools, the major provider of private education in the two cities, are better endowed and 
historically more rooted in the northern port city, whose CathoHc, immigrant population 
dates back to the early Nineteenth Century.^’ 

Interpreting the Magnitude of the Test Score Effects 

Overall, the effects of attending a private school on student test scores are 
moderately large. As can be seen in Table 5, black students who switched to private schools 
scored, after one year, 0.18 standard deviations higher than the students in the control 
group. After two and three years, the size of the effect grew to 0.28 and 0.30 standard 
deviations, respectively, more than a quarter of the difference in test score performances 
between blacks and whites. Continuing evaluations of voucher programs may provide 
information on whether or not these gains can be consolidated and extended. 

Another way of assessing the magnitude of these effects is to compare them to the 
effects observed in an evaluation of a class-size reduction intervention conducted in 
Tennessee, the only other major education reform to be subjected to evaluation by means of 
a randomized field trial. The effects on African Americans of attendance at a private school 
shown here are comparable to the estimated effect of a 7-student reduction in class size. 
According to a recent reanalysis of data from Tennessee, the class-size effect for African 
Americans after two years was, on average, between 7 and 8 percentile points. 

It is also of interest to compare the size of the effects of the voucher intervention 
with the size of the effects reported in a RAND study entitled Improving School Achievementj 
released in August 2000.^^ Identifying the most successful states, Texas and North Carolina, 
which have introduced rigorous accountability systems that involve state -wide testing, the 
study finds what it says are “remarkable’’ one-year gains [in math scores] in these states of 
“as much as 0.06 to 0.07 standard deviation[s] per year” — or 0.18 to 0.21 over three years. 
The three-year effects of the school voucher intervention on black students observed here 
are somewhat larger. 

Cost-Benefit Analysis 

What are these test-score gains for African Americans likely to mean in terms of 
future economic benefits? Duncan Chaplin has estimated the effects of math achievement 
on future earnings.^"* According to one estimate, a 0.30 standard deviation increase in 
average math achievement, if sustained, will yield a 5 percent gain in earnings seven to ten 
years after finishing high school. If an African American student in the control gtoup was 
expected to earn about $30,000 a year in his late twenties, a comparable student who had 
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switched from public to private school would be expected to earn an addidonal $1,500 per 
year. 



This suggests that investments in vouchers might yield a high rate of return for 
African-American families. At the same time, however, it raises the question as to why 
families do not make this investment on their own. If a control-group family had simply 
absorbed the cost of the voucher (on average, about $1,200), even a rough calculation of the 
rate of return suggests that it would be an attractive investment, provided that families can 
borrow moneys at conventional rates. Credit constraints are a possible explanation for the 
decision not to utilize a private school on the part of control-group families. Private lenders 
may be reluctant to make long-term loans at conventional lending rates to low-income 
borrowers who may be high credit risks. If families can borrow the money only at rates 
charged to high risk users of bank cards, then the rate of return on an investment in private 
schooling, though probably stiU positive, would be considerably less attracdve — unless a 
family perceives non-pecuniary benefits of a private educadon. 

Clearly, though, the lower the inidal costs, the more attracdve an investment in 
private schooling becomes. When a voucher reduces the amount that needs to be borrowed 
from around $2,400 a year (a rough esdmate of the average cost of private school tuidon, 
fees, books and uniforms in these cides) to half that amount, families may decide that the 
benefits of an investment in private schooling now outweigh the costs. Perhaps this explains 
why a small voucher induced many low-income families to make the addidonal investment, 
even when members of a similarly situated control group (who did not receive the voucher 
offer) were less likely to do so.^^ 

Addidonal Methodological Consideradons 

The secdon addresses two methodological consideradons. The first involves the 
status of background control variables. In Table 6 we report second and third year results 
for African Americans from stadsdcal models that control not only for inidal test scores (as 
do the analyses in the previous tables) but also for mother's educadon, mother’s employment 
status, family size, and whether or not the family received welfare. The esdmated impacts on 
the test scores of African Americans of switching from a public to a private school in the 
three cides remain almost exacdy the same — 6.4 percentile points in the second year and 
6.7 percendle points in the third. Minor differences are observed when impacts within each 
individual city are esdmated. For instance, when esdmadng effects in New York City in the 
second year without controlling for family background characterisdcs, the impact is 
esdmated to be 4.3 NPR points; when family background controls are added, the impact is 
4.5 NPR points. In Dayton, Ohio, when controls are introduced, the point esdmate drops 
from 6.5 to 5.9 NPR points. And in Washington, D.C., the esdmated impact after two years 
remains 9.2 NPR points. 

The second methodological consideradon concerns the possibility that African 
Americans posted significant effects because they received a more uniform treatment. If the 
black students who used vouchers were dispropordonately concentrated in a small number 
of good private schools, or their peers in the control group concentrated dispropordonately 
in a few bad public schools, the error term in the esdmadon of the private-school effect 
would be smaller for African Americans than for other students. The obvious consequence 
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is to increase the probability that African Americans generate significant effects while 
members of other ethnic groups do not. 

For two reasons, however, we doubt this explanation has much traction. First, the 
size of the standard errors is not all that differentiates the effects for African Africans and 
members of the other ethnic groups. For Latinos in New York and whites in Dayton, the 
point estimates consistendy hover around zero; while for African Americans in both cides, 
the point estimates are quite large. And second, when surveying the private-school 
attendance patterns of students from different ethnic groups, litde evidence emerges to 
support the contention that treatments effects were more uniform for some groups than 
others. African Americans, for the most part, did not attend a relatively smaller number of 
public or private schools than did members of other ethnic groups.^^’ 



Parent Satisfaction 

Most studies have found that families that use vouchers to attend an area private 
school are much more satisfied with their schooling than are families who remain in public 
schools.^^ The second-year results presented in Table 7 confirm these earlier findings. A 
significandy high proportion of private-school parents were “very satisfied’’ with the 
following aspects of their schools: school safety, teaching, parental involvement, class size, 
school facility, student respect for teachers, teacher communication with parents with 
respect to their child’s progress, extent to which child can observe religious traditions, 
parental support for the school, discipline, clarity of school goals, staff teamwork, teaching, 
academic quality, the sports program, and what is taught in school. Thirty-eight percent of 
the private-school parents were very satisfied with the academic quality of the school after 
two year, as contrasted with just 15 percent of the control group. Similarly, 44 percent of 
the private-school parents expressed the highest satisfaction with “what’s taught in school” 
compared with 15 percent of the control group. 

To see whether satisfaction levels are the result of a Hawthorne effect, the 
propensity of individuals to welcome change for its own sake, an index of satisfaction was 
constructed from the items reported above. The positive impact on satisfaction levels in the 
three cities, as measured by this index, was 0.97 standard deviations in the first year, 0.89 in 
the second, and 0.85 in the third year.^® In other words, though overall satisfaction levels 
attenuated slighdy from the first to the second and third years of the evaluation, the 
mediating influence of Hawthorne effects appear minimal. 

Other Voucher Impacts 

Although test-score performance and parental satisfaction are the outcomes of 
greatest interest to most observers, parental surveys provided additional information about 
the impacts of voucher opportunities on selected characteristics of the schools attended by 
students. Significant differences were identified in the school facilities available to students, 
school size, class size, school climate, homework assignment practices, and school 
communication with families. 
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School Facilides 

Public-school expenditures eclipse private-school expenditures. Nadonwide, the 
average private school expenditures per pupil in 1993-94 were esdmated at $3,116, 
considerably less than the $6,653 spent, on average, on public school pupils. In part, this 
disparity is due to the wider array of services public schools provide their students. 
Nonetheless, even when adjustments are made for the kinds of services rendered, public 
schools in New York City, Dayton, and D.C. spend roughly twice as much as private 
schools. 

Per pupil expenditures for both Catholic and public schools were available for 
schools in three boroughs of New York City.‘^ In comparing expenditures, the amount 
spent by New York public schools for all items that did not clearly have a private-school 
counterpart was deducted. Among other things, deducdons were taken for all monies spent 
on transportadon, special educadon, school lunch, other ancillary services, and the cost of 
financing the far-flung bureaucracy that runs the city-wide, borough-wide, and district-wide 
operadons of the New York City public schools. 

Taking all these deducdons from public school expenditures amounted to no less 
than 40 percent of the cost of running the New York City public schools. But even after 
taking aU these and other deducdons, public schools were sdll spending over $5,000 per 
pupil each year, more than twice the $2,400 spent on similar services in New York City’s 
Catholic schools. 

In Washington, D.C., median tuidon at the private schools attended by the 
scholarship students included in the evaluadon was $3,113 in the year 1998-99. The 
average is substandally higher than the median because of the high tuidon charged by a few 
independent schools, such as Sidwell Friends, which charged the Likes of Chelsea Clinton 
over $15,000 per year. Assuming that the rado of tuidon to total educadonal expenditure in 
Washington, D.C. is the same as in the three boroughs in New York City discussed 
previously, private educadonal expenditures, on average, totaled roughly $4,000. Again, 
considering only those services and programs that both public and private schools cover, 
adjusted per-pupil expenditures in Washington public schools reached $8,185, as esdmated 
from data for the 1995-96 school year.^^ 

Much the same patterns emerge in Dayton. In 1998-99, students in the Dayton 
voucher program paid, on average, $2,600 in tuidon, while the Dayton public school system 
spend an adjusted average of $5,528 per pupil. 

Parental reports are consistent with these expenditure data. According to the parents 
surveyed one year into the evaluadon, private schools were less likely to have a library, a 
nurse’s office, a cafeteria, child counselors, and special programs for non— English speakers 
and students with learning problems (see table 8). The greatest difference was for programs 
for non— English speaking students. Forty-three percent of the private-school parents 
reported such a program in their school compared with 71 percent of the control -group 
parents. Similarly, 75 percent of the private-school parents reported their school had a 
nurse’s office, as compared to 94 percent of public-school parents. Public schools are also 
larger. As esdmated by parents, the effect of choosing the private sector was to reduce the 
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average size of the school by 172 students or nearly 40 percent — from an average of 450 
students to 278 students.^^ 

In some instances, either no significant differences were detected or private-school 
parents reported more services. The two groups of parents did not differ in their reports of 
the availability of a gym, a computer laboratory, art and music programs, and special 
programs for advanced learners. Private-school parents, meanwhile, were more likely to say 
their school had individual tutors and an after-school program. 

Despite the more limited financial resources of the private school, parents reported 
that their children had been in classes that had on average three fewer students, 20 as 
compared to 23 in public schools. However, the reduction in class size was only three 
students, considerably less than the amount generally thought to be necessary to achieve 
significant gains from class size reduction.^^ 

School Climate 

In their study of public and private schools, John Chubb and Terry Moe found the 
educational environment of private schools was more conducive to learning than that of 
public schools.^^ They pointed out that public schools are governed by state laws, federal 
regulations, school board requirements, and union-contract obligations that impose multiple 
and not always consistent rules on teachers and principals. Because they must respond to 
numerous legal and contractual requirements, school administrators and teachers focus more 
on rule-compliance than on educational mission, undermining the morale of educators 
whose original objective was to help children learn. 

The problem, Chubb and Moe say, is particularly prevalent in big-city schools. 

Urban private schools operate with greater autonomy, focus more direcdy on their 
educational mission, and, as a result, achieve a higher degree of internal cohesion. To do 
otherwise would jeopardize their survival as a fragile institution dependent upon the annual 
recruitment of new students. As a consequence, principals and teachers in the private sector 
enjoy higher morale. Their interactions with one another and with their students are more 
positive, fostering a more effective learning environment. 

Our findings confirm Chubb and Moe's. If parent reports are accurate, the 
scholarship programs in New York, D.C., and Dayton had a major impact on the daily life of 
students at school. As described in Table 9, public-school parents were more likely to report 
that the following were a serious problem at their school: students destroying property, 
tardiness, missing classes, fighting, cheating, and racial conflict. For example, 32 percent of 
the private-school parents thought that fighting was a serious problem at their school versus 
63 percent of the control group. Thirty three percent of parents perceived tardiness as a 
problem, as compared to 54 for the control gtoup. No more than 22 percent of private- 
school parents but 42 percent of the control group said that destruction of property was a 
serious problem at their school. 

Homework and Parental Communication 
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Thomas Hoffer, Andrew Greeley, and James Coleman have attributed the higher 
level of student performance in private schools to the amount of homework expected of 
students and to the frequency of communication between schools and parents.^^ The 
reports by parents are consistent with their interpretation.*^® Table 10 shows that 72 percent 
of private-school parents reported that their child had at least an hour of homework a day, 
whereas only 56 percent of the control-group parents reported a similar amount of 
homework. Private-school parents were also more likely to say the homework was 
appropriate for their child. Seventy- two percent of the control-group parents gave this 
response, as compared to 90 percent of private-school parents. 

Compared with control-group parents, parents of students in private schools also 
said that they received more communication from their school about their child. The results 
presented in Table 1 1 indicate that a higher percent of private-school parents versus control- 
group parents reported: being more informed about student grades halfway through the 
grading period; being notified when their child is sent to the office the first time for 
disruptive behavior; parents speaking to classes about their jobs; regular parent-teacher 
conferences; parents participating in instruction; parents receiving notes about their child 
from the teacher; parents receiving a newsletter about what is going on in school; and regular 
parent-teacher conferences. 



Conclusions 

Randomized field trials are the best available tool for detecting the effects of an 
educational intervention, because random assignment to test and control groups assures that 
all significant effects may be attributed to the intervention, not to the students’ initial abilities 
or their family backgrounds. Nonetheless, when interpreting the findings from the 
evaluation of any one program in a particular city, generalizations to a larger universe are 
problematic. Conditions specific to that place or minor fluctuations in testing conditions 
might skew results in one direction or another. 

Still, when similar results emerge from evaluations of school voucher programs in 
three sites in different parts of the United States, they provide a stronger basis for drawing 
conclusions and generalizing to a larger context. Thus, the average impact across the three 
sites may provide a reasonable estimate of the likely initial impact of a school voucher 
initiative elsewhere. 

In the three cases, taken together, we found effects of school vouchers only on the 
average test performance of students from African American backgrounds. Black students 
who switched from public to private schools in the three cities scored after two years, on 
average, approximately 6.3 percentile points higher on the Iowa Test of Basic Skills than 
comparable blacks who remained in public schools. After three years, private school 
attendance in two cities had an impact of 6.6 percentile points, an impact of 0.30 standard 
deviations. 

At this point we do not know why the gains from switching to a private school are 
evident for black students after two and three years, but not for students from other ethnic 
backgrounds. However, parents in New York reported that private schools are smaller in 
size, maintain a better disciplinary climate, ask students to do more homework, maintain 
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closer communication with families, and have somewhat smaller classes (about 3 fewer 
pupils). 



One must qualify any generalizations from the results of this pilot program to a 
large-scale voucher program that would involve all children in a large urban school system. 
Only a small fraction of low-income students in these three cities’ schools were offered 
vouchers, and these voucher students constituted only a small proportion of the students 
attending private schools in these cities. A much larger program could conceivably have 
quite different program outcomes. 

Still, slighdy larger voucher programs initially directed at low-income families would 
attract those families with the greatest interest in exploring an educational alternative, exacdy 
the group that applied for a voucher in these three cides. Thus, positive consequences of 
school choice reported herein may prove encouraging to those who seek to extend and 
expand school choices for low-income, inner-city families, while negative findings may 
indicate problems that need to be addressed. It is hoped that addidonal careful research will 
accompany larger programs established by private philanthropists and/or public authorides.^ 
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TABLE 1 : The Impact in Three Cities of Switching to a Private 
School on Test Score Performances 



Test Score Performance 


Year 1 

(Percentiles) 


Year 2 
(Percentiles) 


Year 3 
(Percentiles) 


African Americans 














Overall 


3.9* 


(2.0) 


6.3*** 


(2.5) 


6.6** 


(2.8) 


Math 


6.1*** 


(2.4) 


6.1** 


(3.1) 


4.2* 


(2.2) 


Reading 


2.1 


(2.4) 


5.9** 


(2.9) 


4.2 


(3.5) 


All Other Ethnic Groups 














Overall 


-1.6 


(2.4) 


-1.4 


(2.9) 


-3.5 


(2.7) 


Math 


-2.5 


(3.1) 


-2.6 


(3.9) 


-2.7 


(3.3) 


Reading 


-0.7 


(2.5) 


-0.2 


(3.0) 


-4.2 


(2.9) 



** significant at the .05 level, two tailed test; * .10 level. Figures represent the average impact of 
switching to a private school on test score performance scores in New York, D.C., and Dayton. 
Averages are based upon effects observed in the three cities weighted by the inverse of the variance of 
the point estimates. Standard errors reported in parentheses. For African Americans, the unweighted 
average effects after one year are 2.7 overall, 4.8 in math, and 0.6 in reading; after two years, the 
unweighted average effect sizes are 6.6 overall, 6.5 in math, and 6.8 in reading. 
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TABLE 2: Impact in New York of Switching to a Private School 
on Test Score Performance 



Test Score 
Performance 


Year 1 
(Percentiles) 


(N) 


Year 2 
(Percentiles) 


(N) 


Year 3 
(Percentiles) 


(N) 


African Americans 




















Overall 


5.4*** 


(1.5) 


622 


4 . 4 ** 


(2.0) 


497 


9,2*** 


(2.4) 


519 


Math 


6.9*** 


(1.8) 


622 


4 . 1 * 


(2.5) 


497 


11,8*** 


(2.9) 


519 


Reading 


4,0** 


(1.8) 


622 


4 . 5 ** 


(2.3) 


497 


6.7** 


(2.9) 


519 


All Other Ethnic Groups 




















Overall 


-2.2 


(1.8) 


812 


-1.5 


(2.2) 


699 


-3.5 


(2.4) 


729 


Math 


-3.2 


(2.3) 


812 


-3.2 


(2.9) 


699 


-2,5 


(2.9) 


729 


Reading 


-1.2 


(1.9) 


812 


0.2 


(2.3) 


699 


-4.4* 


(2.6) 


729 



* significant at .10 level, 2-tailed test; ** .05 level; *** .01 level. Weighted two-stage least squares regressions performed; 
treatment status used as instrument. Standard errors reported in parentheses. All models control for baseline test scores and 
lottery indicators. Impacts expressed in terms of national percentile rankings. When using bootstrapped standard errors, the 
year 2 math score is not statistically significant; the significance levels of all other estimates remain the same when significance 
levels are estimated using the bootstrap technique. See Robert Stine, 1990. "An Introduction to Bootstrap Methods: Examples 
and Ideas" in J, Fox and J. S. Long, eds,. Modem Methods of Data Analysis , p, 325-373. Newbury Park. CA: Sage 
Publications. Bradley Effron, 1982. "The Jackknife, the Bootstrap and Other Resampling Plans." Philadelphia, PA: Society for 
Industrial and Applied Mathematics. 
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TABLE 3: Impact in Dayton of Switching to a Private School on Test Score Performance 



Test Score Performance 


Year 1 
(Percentiles) 


(N) 


Year 2 
(Percentiles) 


(N) 


African Americans 














Overall 


3.3 


(3.5) 


296 


6.5* 


(3.7) 


273 


Math 


0.4 


(4.0) 


296 


5.3 


(4.3) 


273 


Reading 


6.1 


(4.2) 


296 


1 . 6 * 


(4.2) 


273 


All Other Ethnic Groups 














Overall 


1.0 


(6.4) 


108 


-0.2 


(9.0) 


96 


Math 


-0.8 


(7.5) 


108 


0.0 


(10.7) 


96 


Reading 


2.8 


(7,1) 


108 


-0.4 


(9.9) 


96 



* significant at .10 level, 2-tailed test; ** .05 level; *** .01 level. Weighted two-stage least squares 
regressions performed; treatment status used as instrument. All models control for baseline test scores. 
Impacts expressed in terms of national percentile rankings. Standard errors reported in parentheses. 
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TABLE 4: Impact in D.C. of Switching to a Private School on Test Score Performance 



Test Score Performance 


Year 1 

(Percentiles) 


(N) 


Year 2 
(Percentiles) 


(N) 


Year 3 
(Percentiles) 


(N) 


African Americans 




















Overall 


-0,9 


(2.8) 


891 


9,2*** 


(2.9) 


668 


-1,9 


(4.4) 


656 


Math 


7,3** 


(3.3) 


891 


10.4*** 


(3.4) 


668 


0,9 


(1.9) 


656 


Reading 


-9,0** 


(3.7) 


891 


8,0*** 


(3.4) 


668 


-4,6 


(5.4) 


656 


All Other Ethnic Groups 




















Overall 


7.4 


(8.7) 


39 


-0.1 


(9.8) 


42 


-1.8 


(13.3) 


31 


Math 


8.5 


(10.7) 


39 


7.3 


(13.4) 


42 


-9.5 


(15.4) 


31 


Reading 


6,3 


(12.7) 


39 


-7.6 


(10.1) 


42 


5,9 


(18.7) 


31 



* significant at ,10 level, 2-tailed test; ** .05 level; *** ,01 level. Weighted two-stage least squares regressions performed; 
treatment status used as instrument. Standard errors reported in parentheses. All models control for baseline test scores; in year 1, 
models also control for initial testing session. Impacts expressed in terms of national percentile rankings. 
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Table 5: Size of the Effects of Switching to a Private School on African Americans’ Overall 

Test Score Performances 



Test Score Performance 


Effect Size 
Year One 

(Standard Deviations) 


Effect Size 
Year Two 

(Standard Deviations) 


Effect Size 
Year Three 
(Standard Deviations) 


Overall 


0.18 


0.28 


0.30 


Math 


0.28 


0.28 


0.18 


Reading 


0.08 


0.23 


0.16 



Figures represent the unweighted average impact of switching to a private school on test scores in New York, 
D.C., and Dayton expressed in standard deviations. 
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Table 6: Estimated Effects after Two Years of Switching from a Public to a 
Private School on African Americans’ Combined Test Scores, With and 
Without Controls for Family Background Characteristics 





Private- School Impact, 


Private-School Impact, 




Original Results 


Controlling for Family 








Background 


Three-City Average Impact 


6.3*** 


(2.5) 


6.4** 


(2.5) 


New York City 


4.4** 


(2.0) 


4.5** 


(2.0) 


Dayton, OH 


6.5* 


(3.7) 


5.9 


(3.8) 


Washington, D.C. 


9.2*** 


(2.9) 


9.2*** 


(2.8) 



* significant at .10 level, 2-tailed test; ** .05 level; *** .01 level. P-values reported in brackets. 

Weighted two-stage least squares regressions perfomied; treatment status used as instrument. 
All models control for baseline test scores, mother’s education, employment status, whether or 
not the family receives welfare, and family size (missing case values for demographic variables 
estimated by imputation); NY model also includes lottery indicators. Impacts expressed in 

terms of national percentile rankings. Average three-city impact is based on effects observed in 
the three cities weighted by the inverse of the standard errors of the point estimates. 



TABLE 7: Parent Satisfaction with School, Two Years after Beginning of 
Voucher Programs (Percent "Very Satisfied") 



Parent Satisfaction with: 


Switched to 
Private School 

d)(a) 


Public School 
Control Group 


Year Two Programmatic 
Impact 
(3)<'> 


Discipline 


35 


14 


22*’* 


Academic quality 


38 


15 


23*** 


Teacher skills 


43 


17 


26*** 


Student respect for teachers 


40 


16 


24*** 


What taught in school 


44 


15 


29*** 


Teaching values 


36 


14 


23*** 


Class size 


32 


12 


20*** 


Clarity school goals 


34 


14 


20*** 


Parental involvement 


30 


15 


15** 


Ability observe relig. traditions 


37 


8 


29*** 


School safety 


44 


16 


27*** 


Staff teamwork 


34 


13 


21*** 


Location 


40 


33 


7 


Teacher-parent relations 


43 


18 


25*** 



These figures represent the average results for New York City, Dayton, and D.C.. Observations 
from each city are weighted by the inverse of their variance. * significant at . 1 level, two-tailed 
test; significant at .05 level; significant at .01 level. 

Those who were offered a scholarship and as having used their scholarship to attend a private 
school. 

Those in the control group who would have used a scholarship had they been offered one. 

Estimated impact of participation in the program, using a two-stage least squares model. 
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Table 8: Size and Quality of School Facilities, One Year after Beginning of 
Voucher Programs (Percentages) 



Parental Reports: 


Switched to 
Private School 
(1) 


Public School 
Control Group 
(2) 


Programmatic 

Impact 

( 3 ) 


Average school size 


278 


450 


-172*** 


Average class size 


20 


23 


-3*** 


Percentage satisfied with school 
facilities 


28 


9 


19*** 


Percentage with the following 
resources 








Special programs for non- 
English speakers 


43 


71 


-28*** 


Nurses’ office 


75 


94 


-19*** 


Special programs for learning 
disabled 


67 


81 


-14*** 


Cafeteria 


86 


96 


-10*** 


Child counselor 


77 


85 


-8*** 


Library 


92 


96 


-5** 


Gym 


88 


88 


0 


Special programs for advanced 
learners 


59 


58 


1 


Arts program 


82 


81 


1 


Computer lab 


86 


84 


2 


Music program 


88 


84 


4 


After-school program 


91 


86 


6** 


Individual tutors 


70 


54 


16*** 



See notes to Table 7. 
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Table 9: Parents’ Perceptions of School Climate, One Year after Beginning of 
Voucher Programs (Percentages) 



Switched to Public School Programmatic 

Private School Control Group Impact 



ill m Q1 



Parents report as serious 
problem: 








Fighting 


32 


63 


-3f** 


Kids missing class 


26 


48 


-22*** 


Tardiness 


33 


54 


-21*** 


Kids destroying property 


22 


42 


-20*** 


Cheating 


26 


39 


-13*** 



See notes to Table 7. 
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Table 10: Homework, One Year after Beginning of Voucher Programs 

(Percentages) 





Switched to 
Private School 
(1) 


Public School 
Control Group 
(2) 


Programmatic 

Impact 

(3) 


Parents report: 








Child has more than one hour 
of homework 


72 


56 


16”' 


Difficulty of homework 
appropriate for child 


90 


72 


18'" 



See notes to Table 7. 
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Table 11: School Communication with Parents, One Year after Beginning of 
Voucher Programs (Percentages) 



Switched to Private Public School Programmatic 

Parent Reports School Control Group Impact 

(D (2) (3) 



Parents receive newsletter 


88 


68 


20*** 


Parents participate in instruction 


68 


50 


18*** 


Notified of disruptive behavior 


91 


77 


14*** 


Parents receive notes from 
teacher 


93 


78 


14*** 


Parents speak to classes about jobs 


44 


33 


11** 


Parents regularly informed about 
student grades 


93 


84 


9** 


Parent open houses held at school 


95 


90 


5** 


Regular parent-teacher conferences 


95 


90 


5** 



See notes to table 7. 
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Appendix A: Response Rates 

To promote high response rates, voucher program operators either required or 
strongly urged recipients to pardcipate in tesdng sessions if they wished to have their 
voucher renewed for the next school year. In addidon, evaluadon teams offered financial 
incendves and new opportunides to win a voucher to encourage members of the control 
group and members of the treatment group who remained in public schools to return for 
follow-up tesdng. Sdll, substandal numbers of students were not tested at the end of one, 
two, and three years. 

Response rates were 100 percent at baseline, because families and students were not 
entered into the lottery unless they provided baseline informadon. Response rates after one 
year were 82 percent in New York, 56 percent in Dayton and 63 percent in Washington, 
D.C. After year two, response rates in the three cides were 66 percent 49 percent, and 50 
percent, respecdvely. After three years, response rates were 67 percent in New York and 60 
percent in Washington, D.C. Response rates were similar for treatment and control groups 
in all three cides. The largest difference was in New York City in the second year, where the 
treatment group's response rate was 7 points higher than the control-group rate.^^ 

Comparisons of baseline test scores and background characterisdcs reveal only 
minor differences between respondents and nonrespondents in all three cides. Table A-1 
presents, for example, baseline data on respondents and nonrespondents in the treatment 
and control groups after two years in the three cides; separate comparisons for African 
Americans are included in Table A-2. Some differences in race, welfare, and religious 
orientadon were detected, but they point in different direcdons in different cides and do not 
appear to systemadcally produce a more advantaged group of respondents in the treatment 
group nor a pardcularly disadvantaged control group. In all three cides, inter-group 
differences in test scores, religious idendficadon, residendal mobility rates, church 
attendance, and family size were essendally nonexistent. 

To adjust for the bias associated with nonresponse, in each year and city we 
generated weights for parents and students in the treatment and control groups. Because 
those invited to pardcipate in the follow-up studies had provided informadon about their 
family characterisdcs at baseline, it was possible to calculate the probability that each 
pardcipant in the baseline survey would attend a follow-up session. To do so, we esdmated 
simple logit regressions that used a set of variables assembled from baseline surveys to 
predict the likelihood that each student would attend a follow-up session. Covariates 
included mother’s educadon, employment status, marital status, and religious affiliadon; 
family size; whether the family received welfare benefits; whether the student was African 
American; the student’s baseline math score; whether the student had a learning disability; 
and whether the student had experienced disciplinary problems.^ 

To allow for as much flexibility as possible, separate logit models were esdmated for 
treatment and control group members. For iUustradve purposes, table A-3 reports the results 
in Washington after two years. Similar results were obtained for other cides and other years."^ 
For the most part, the family and student characterisdcs had a similar impact on response 
rates for both treatment and control group members. Catholics were less likely to attend 
follow-up sessions, as were mothers who were employed full-dme or were married. Larger 
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families were more likely to attend follow-up sessions, as were African American families 
and families of students with disciplinary problems. Mother’s education, welfare benefits, 
and math scores had a small or insignificant impact for both treatment and control group 
members. The most striking difference between the two models concerned students with 
learning disabilities. While learning disabled students in the treatment group were 
significandy more likely to attend follow-up sessions, such students in the control group 
were significandy less likely to attend follow-up sessions. 

The models generated a set of predicted values that represent the probability that 
individuals, given their baseline characterisdcs, would attend the follow-up session. The 
weights are simply the inverse of these predicted values, that is, 



' F{^/3 ) 

where F( ) is the model’s normal cumuladve distribudon funcdon. The range of possible 
values for Wj was then capped so that the highest weight was four dmes the value of the 
lowest. (This restricdon affected only a handful of observadons.) The weights then were 
rescaled so that the sum of the weights equaled the sum of the total number of actual 
observadons. 

To generate the weights we could use only observable characterisdcs as recorded in 
parent surveys. To the extent that there were unmeasured or unobservable characterisdcs 
that encouraged some families, but not others, to attend follow-up sessions, the weights may 
not have eliminated the bias associated with nonresponse. However, in order for response 
bias to explain our findings, three condidons would have to hold. First, respondents would 
need to differ from nonrespondents on an unmeasured factor that influenced test 
performance. Second, the difference would have to be larger for one gfoup (treatment or 
control) than for the other. Third, the difference would have to hold for black students but 
not for students of other ethnic groups. While we cannot rule out the possibility that all 
three condidons existed in our study, we find it unlikely enough to be reasonably confident 
that response bias did not ardficially generate the results we report. 

It is possible that change in academic performance over dme rather than baseHne 
characterisdcs affected the likelihood that different subgroups within the treatment and 
control groups would attend subsequent tesdng sessions. If treatment group families that 
did not benefit from vouchers dropped out of the study while control gtoup families that 
were suffering most in public schools continued to attend follow-up sessions consistendy, 
then observed impacts may be somewhat inflated. 

Three questions deserve consideration. Did gains in test scores from baseline to year 
one (two) decrease the probability that members of the control group would attend the year 
two (three) tesdng session? Did gains increase the probability that members of the 
treatment group would attend the year two (three) tesdng session? Were the differences in 
observed impacts on response rates for the treatment and control groups statistically 
significant? 
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Table A-4 estimates a series of logistic regressions that answer these questions. The 
dependent variable idendfies whether a student attended the year two (three) follow-up 
session. The covariates include baseline math and reading test scores, the change in the total 
test score from baseline to year one (two), and the change interacted with treatment status. 
Separate models were run for African Americans and members of other ethnic groups. At 
the bottom of each column we report the probability that we can reject the following three 
nuU hypotheses: (1) changes in test scores have a statistically insignificant effect on 
attendance at subsequent testing sessions for the control group; (2) the effect for the 
treatment group is statistically insignificant; and (3) the differences in observed effects for 
the two groups are not statistically significant. 

On the whole, the signs of the coefficients are in the expected direction. Gains in 
test scores from baseline to years one and two increased the probability that members of the 
treatment group attended the subsequent testing session and decreased the probability for 
members of the control group. The only models that generated statistically significant 
impacts, however, were for African Americans in New York after three years and for African 
Americans in Dayton after two years. None of the observed impacts for Hispanics were 
statistically significant in any year or city. 

The model that predicts year three attendance for African Americans in New York 
City generated the largest effects. Holding aU variables at their means, the model predicted 
that 83 percent of the students who attended the year two session would attend the year 
three session. An increase of 10 NPR points from baseline to year two translated into a 3 
percentage point drop in the probability that a control group member would attend the year 
three testing session and a 1 percentage point increase in the probability that a member of 
the treatment group would attend the year three follow-up session. Unless weighting 
adjusted for these differences, this response pattern may have marginally contributed to the 
positive estimate of voucher impacts on test scores. 

In New York City, eighty-two African American students who had attended the year- 
two testing session failed to show up in year three. The data presented above suggest that 
those individuals consisted disproportionately of control group members whose scores 
decreased from baseline to year two and treatment group members whose scores increased, 
possibly inflating the estimated impact of attending a private school. To further explore 
their influence on estimated impacts, we imputed year three test scores for those individuals 
based on their treatment status, baseline test scores, test score changes between baseline and 
year two, and the year three weights. While the observed impacts do drop in magnitude, 
they remain statistically significant. When we examined only those African American 
students who attended the year three follow-up session, the estimated impact of being 
offered a voucher (which is different from the impact of actually attending a private school, 
as discussed in chapter 2) at year three was 5.4 NPR points, with 515 observations and a /- 
statistic of 3.7.^ When we looked at the same population but then imputed year three test 
scores for those students who showed up in year two but not in year three, the si 2 e of the 
estimated impact of being offered a voucher dropped to 4.6 NPR points, with a /-statistic of 
3.0.' 



Another way of estimating the effects of response rates on outcomes is to distinguish 
between earlier and later respondents. Not aU participants came to the first testing session 
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to which they were invited. Given that we know the dates when students came in for 
testing, we can generate exact esdmates of the impact of attending a private school for 
smaller response rates. In year one in New York City, for instance, we had an 82 percent 
response rate. By successively dropping the portion of students who attended later testing 
sessions, we can readily calculate the impacts for lower response rates. 

If observed positive impacts derive from imperfect response rates, we should expect 
the estimated impact of attending a private school to increase as response rates decline. 
Presumably, those students who benefit most from treatment should come earlier to the 
testing sessions, along with those students in the control group who were performing most 
poorly in public schools. Impacts of attending a private school, then, should be quite large 
for lower response rates. The differences between the two groups, however, should 
attenuate (and may actually switch signs) as response rates increase. 

Table A-5 reports the estimated impact of attending a private school for African 
American students for variable response rates. In each row, the first column represents the 
esdmated impact for the full sample of African American students who attended tesdng 
sessions. Subsequent columns provide esdmates of impacts for lower response rates, based 
on when students came in for tesdng. 

As can be seen in table A-5, the New York City esdmates remained remarkably 
stable for different response rates. Had we stopped tesdng students in Year I after the first 
30 percent of the sample showed up, we would have recovered almost exacdy the same 
findings that we did after another 52 percent pardcipated — the point esdmate for the first 30 
percent of students to be tested was 5.7 percentile points, and it was 5.4 for the fuU sample. 
In New York City in years two and three, rather than increasing as response rates declined, 
the esdmated impacts decreased. Moving from a 30 percent response rate to a 66 percent 
rate, the esdmated test score impact of attending a private school increased by roughly 1 
NPR point and became stadsdcally significant. From these findings, at least, there is litde to 
suggest that we would have observed significandy different impacts had we managed to test 
a greater number of students in the treatment and control groups. Observed impacts 
remained quite steady over the course of tesdng sessions conducted each year. 
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Table A1 : Baseline Characteristics of Respondents and Non-Respondents in 
Treatment and Control Groups in Year II 





TREATMENT 


CONTROL 


Attended Yr II 


Didn’t Attend Yr II 


Attended Yr II 


1 Didn’t Attend Yr II 


New York City 










Pet African American 


42.4 


48.3 


41.4 


47.2 


Pet Welfare Recipients 


53.2 


64.5 


59.4 


62.3 


Pet Catholic 


54.7 


46.4 


53.7 


43.2 


Pet Protestant 


34.3 


39.4 


35.0 


38.8 


Ave Overall Test Scores 


20.1 


19.5 


22.8 


22.6 


Ave Family Size 


2.6 


2.6 


2.4 


2.9 


Ave Residential Mobility 


3.7 


3.6 


3.7 


3.7 


Ave Church Attendance 


3.6 


3.3 


3.4 


3.5 


Ave Mother’s Education 


2.4 


2.4 


2.4 


2.5 


Dayton 










Pet African American 


74.0 


65.2 


71.9 


69.3 


Pet Welfare Recipients 


16.7 


13.8 


16.2 


16.7 


Pet Catholic 


5.8 


14.0 


13.4 


18.1 


Pet Protestant 


65.2 


58.1 


64.6 


56.9 


Ave Overall Test Scores 


26.3 


26.3 


27.2 


26.2 


Ave Family Size 


3.0 


3.0 


3.0 


3.1 


Ave Residential Mobility 


3.4 


3.3 


3.3 


3.6 


Ave Church Attendance 


3.4 


3.3 


3.6 


3.7 


Ave Mother’s Education 


5.6 


5.4 


5.3 


5.6 


D.C. 










Pet African American 


90.4 


92.1 


90.9 


92.1 


Pet Welfare Recipients 


38.0 


34.1 


32.1 


30.3 


Pet Catholic 


15.5 


12.6 


16.0 


13.8 


Pet Protestant 


72.7 


69.9 


65.6 


70.6 


Ave Overall Test Scores 


26.5 


26.4 


26.9 


26.7 


Ave Family Size 


3.1 


3.1 


3.3 


3.0 


Ave Residential Mobility 


3.4 


3.5 


3.5 


3.4 


Ave Church Attendance 


3.7 


3.5 


3.7 


3.7 


Ave Mother’s Education 


5.4 


5.0 


5.3 


5.2 



Averages refer to the unweighted mean scores of responses on the parent surveys. Mother’s education was scaled slightly 
differently in New York City than in Dayton and Washington, D.C., making intercity comparisons on that item 
inappropriate. 
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Table A2: Baseline Characteristics of African American Respondents and Non- Respondents in 

Treatment and Control Groups in Year Two 





TREATMENT 


CONTROL 


Attended Yr Two 


Didn’t Attend Yr Two 


Attended Yr Two 


Didn’t Attend Yr Two 


New York City 










Pet Welfare Recipients 


55.3 


63.5 


65.8 


65.8 


Pet Catholic 


17.8 


18.0 


19.5 


8.6 


Pet Protestant 


66.9 


67.3 


66.8 


65.2 


Ave Overall Test Scores 


20.6 


19.1 


21.2 


23.8 


Ave Family Size 


2.6 


2.8 


2.5 


3.1 


Ave Residential Mobility 


3.8 


3.7 


3.7 


3.7 


Ave Church Attendance 


3.4 


3.4 


3.2 


3.3 


Ave Mother’s Education 


2.5 


2.5 


2.5 


2.6 


Dayton 










Pet Welfare Recipients 


15.9 


15.0 


17.9 


20.7 


Pet Catholic 


7.6 


12.4 


8.7 


4.7 


Pet Protestant 


66.1 


61.9 


76.5 


69.8 


Ave Overall Test Scores 


24.3 


21.6 


23.2 


22.0 


Ave Family Size 


2.7 


2.6 


2.8 


2.9 


Ave Residential Mobility 


3.4 


3.2 


3.3 


3.6 


Ave Church Attendance 


3.8 


3.7 


3.9 


4.0 


Ave Mother’s Education 


6.1 


5.8 


5.5 


5.4 


D.C. 










Pet Welfare Recipients 


38.8 


31.6 


34.7 


27.9 


Pet Catholic 


13.0 


15.1 


13.7 


16.0 


Pet Protestant 


76.2 


69.4 


67.8 


67.7 


Ave Overall Test Scores 


26.2 


28.3 


26.1 


28.4 


Ave Family Size 


3.1 


3.0 


3.3 


2.9 


Ave Residential Mobility 


3.5 


3.5 


3.5 


3.5 


Ave Church Attendance 


3.7 


3.6 


3.7 


3.8 


Ave Mother’s Education 


5.4 


5.2 


5.3 


5.4 



Averages refer to the unweighted mean scores of responses on the parent surveys. Mother’s education was scaled 
slightly differently in New York City than in Dayton and Washington, D.C., making intercity comparisons on that item 
inappropriate. 
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Table A3: Logit Estimates Used to Construct Weights For Treatment and Control 
Groups in Washington, D.C. in Year II 





Treatment Group 


Control Group 


Family Characteristics 






Catholic 


-0.5* 


-0.8*** 


Family size 


0.2** 


0.2** 


Employment Status 


-0.6** 


-0.1 


Married 


-0.6*** 


-0.3 


Mother’s Education 


0.0 


-0.1** 


Welfare 


-0.3 


0.2 


African American 


0.8*** 


0.6*** 


Student Characteristics 






Learning Disabled 


0.7** 


-1.0** 


Disciplinary Problems 


0.7** 


0.7** 


Math 


-0.0 


-0.0** 


Constant 


-1.1** 


-0.6 


Pseudo 


.07 


.07 


Log likelihood 


-353.11 


-479.83 


(N) 


580 


866 



*/? < .10, two-tailed test; < .05; ***/? < .01. The dependent variable is coded 1 if the student attended 
the Year II follow-up session in Washington, D.C., and 0 otherwise. The treatment group consists of all 
students who were offered a voucher and participated in the baseline study; the control group consists of all 
students who were not offered a voucher. 




36 



36 



Table A4: Effect of Change in Test Scores From Baseline to Year I and II on the 
Likelihood That Students Attend Subsequent Testing Sessions 





NEW YORK CITY 


D.C. 


DAYTON 


Year II Attendance 


Year 111 Attendance 


YII Attend 


Year 11 Attendance 




Blacks 


Latinos 


Blacks 


Latinos 


Blacks 


Blacks 


Whites 


Yl-Baseline 


-0.005 


-0.001 






0.002 


-0.012 


0.001 


Y2-Baseline 






-0.023* 


-0.004 








(Yl-B)*Treat 


0.0037 


0.006 






0.002 


0.023* 


-0.004 


(Y2-B)*Treat 






0.031* 


0.004 








Baseline Math 


-0.003 


0.012* 


0.007 


0.002 


-0.001 


0.001 


-0.003 


Baseline Reading 


0.007 


-0.004 


0.006 


-0.008 


-0.001 


0.003 


0.004 


(N) 


623 


709 


497 


612 


891 


298 


108 


Log-likelihood 


-355.81 


-355.68 


-212.40 


-191.13 


-580.55 


-189.52 


-70.39 


Pseudo-R^ 


0.00 


0.01 


0.02 


0.00 


0.00 


0.01 


0.00 


Pfor Ho:B,=0 


0.63 


0.91 


0.09 


0.76 


0.79 


0.21 


0.94 


P for Hq: Bi+ B2=0 


0.90 


0.57 


0.50 


0.96 


0.50 


0.28 


0.79 


P for Hq: B2=0 


0.77 


0.46 


0.07 


0.78 


0.79 


0.07 


0.79 



*p < .10, two-tailed test; **p < .05; < .01. Logit regression models performed on unweighted data. Yl-Baseline refers 

to the change in the total math and reading test scores from baseline to Year 1; Y2-Baseline refers to change from baseline to 
Year II. (Yl-B)*offered voucher is an interaction term between one variable that is the difference between Year 1 and 
baseline test scores and another variable that indicates whether a student was offered a voucher. The dependent variable is 
coded 1 if the student attended either the second- or third-year follow-up session. 
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Table A5: Estimated Impacts of Attending a Private School for African Americans 
in New York City for Variable Response Rates 





Percentage of Respondents Attending Follow-Up Sessions 


30% 


40% 


56% 


60% 


66% 


70% 82% 


Year I Impact 




4 4*** 


4 2*** 


4.8*** 


5.0*** 


^ 2*** ^ 4*** 


Year II Impact 


3.6 


2.7 


4 4** 


3.2* 


4.3** 




Year III Impact 


4.2 


6 9*** 


7 1*** 


8.3*** 


9 2*** 





* significant at .10 level, two-tailed test; ** .05 level; *** .01 level. Weighted two-stage least squares regressions 
performed; treatment status used as instrument. Differential response rates calculated by including in the analysis 
only the relevant percentage of students to initially attend testing sessions. 
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Appendix B: The Effects of the Offer of a Voucher 

Tables B-1, B-2 and B-3 report estimated effects of a voucher offer on student test 
score performance in each city. These intention-to-treat effects are smaller than actual 
treatment of effects, because many students who were offered vouchers did not make use of 
them, and others who were not offered vouchers found alternative ways of financing a 
private education. The percentages using a voucher in each city is reported in the text of this 
chapter. 
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TABLE B 1 : Impact in New York of Being Offered a Voucher 
on Test Score Performance 



Test Score Perfomiance 


Year 1 
(Percentiles) 


(N) 


Year 2 
(Percentiles) 


(N) 


Year 3 
(Percentiles) 


(N) 


African Americans 




















Overall 


4.5** 


(1.2) 


622 


3.3** 


(1.5) 


497 


5.5*** 


(1.4) 


519 


Math 


5.7*** 


(1.5) 


622 


3.1* 


(1.9) 


497 


7.0*** 


(1.7) 


519 


Reading 


3.3** 


(1.5) 


622 


3.4** 


(1.7) 


497 


4.0** 


(1.7) 


519 


All Other Ethnic Groups 




















Overall 


-1.3 


(1.3) 


812 


-1.0 


(1.5) 


699 


-2.3 


(1.5) 


729 


Math 


-2.4 


(1.7) 


812 


-2.2 


(2.0) 


699 


-1.7 


(1.9) 


729 


Reading 


-0.9 


(1.4) 


812 


0.1 


(1.6) 


699 


-2.9* 


(1.7) 


729 



* significant at .10 level, 2-tailed test; ** .05 level; *** .01 level. Weighted OLS regressions performed. All models 
control for baseline test scores and lottery indicators. Impacts expressed in terms of national percentile rankings. 
Standard errors reported in parentheses. When using bootstrapped standard errors, the year 2 math score is not 
statistically significant; the significance levels of all other estimates remain the same when significance levels are 
estimated using the bootstrap technique. See Robert Stine, 1990. "An Introduction to Bootstrap Methods: Examples and 
Ideas" in J. Fox and J. S. Long, eds., Modem Methods of Data Analysis, p. 325-373. Newbury Park. CA: Sage 
Publications. Bradley Effron, 1982. "The Jackknife, the Bootstrap and other Resampling Plans." Philadelphia, PA: 
Society for Industrial and Applied Mathematics. 
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TABLE B2: Impact in D.C. of Being Offered a Voucher on Test Score Performance 



Test Score Performance 


Year 1 
(Percentiles) 


(N) 


Year 2 
(Percentiles) 


(N) 


Year 3 
(Percentiles) 


(N) 


African Americans 




















Overall 


-0.3 


(1.1) 


891 


3.8*** 


(1.2) 


668 


-0.5 


(1.2) 


656 


Math 


2.9** 


(1.3) 


891 


4.3*** 


(1.4) 


668 


0.3 


(1.4) 


656 


Reading 


-3.6** 


(1.5) 


891 


3.3** 


(1.4) 


668 


-1.3 


(1.5) 


656 


All Other Ethnic Groups 




















Overall 


4.7 


(5.6) 


39 


-0.1 


(5.6) 


42 


-0.9 


(6.6) 


31 


Math 


5.5 


(7.2) 


39 


4.1 


(7.4) 


42 


-4.7 


(7.7) 


31 


Reading 


4.0 


(8.0) 


39 


-4.3 


(5.7) 


42 


2.9 


(9.1) 


31 



* significant at .10 level, 2-tailed test; ** .05 level; *** .01 level. Weighted OLS regressions performed. Standard 
errors reported in parentheses. All models control for baseline test scores; in year 1 , models also control for initial 
testing session. Impacts expressed in terms of national percentile rankings. 




TABLE B3: Impact in Dayton of Being Offered a Voucher 
on Test Score Performance 



Test Score Performance 


Year 1 
(Percentiles) 


(N) 


Year 2 
(Percentiles) 


(N) 


African Americans 














Overall 


1.9 


(2.0) 


296 


3.5* 


(2.0) 


273 


Math 


0.2 


(2.3) 


296 


2.8 


(2.3) 


273 


Reading 


3.5 


(2.4) 


296 


4.1* 


(2.3) 


273 


All Other Ethnic Groups 














Overall 


0.7 


(4.1) 


108 


-0.1 


(4.0) 


96 


Math 


-0.5 


(4.8) 


108 


0.0 


(4.7) 


96 


Reading 


1.8 


(4.5) 


108 


-0.2 


(4.4) 


96 



* significant at .10 level, 2-tailed test; ** .05 level; *** .01 level. Weighted OLS regressions performed. 
Standard errors reported in parentheses. All models control for baseline test scores. Impacts expressed in 
terms of national percentile rankings. 
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Elacqua and Matthew Charles for their key roles in coordinating data collection efforts. 
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methodology, analyses of data, reported findings, and interpretations of findings are the sole 
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WSF, PACE, or of any foundation providing support for this research. 

^ R. J. Bridge and J. Blackman^ Study of JUtematives in A^merican Education: Vol. 4. Family 
Choice in Education (Santa Monica, CA: Rand Corporation, 1978); Richard Elmore, "Choice as 
an Instrument of Public Policy: Evidence from Education and Health Care," in W. Clune & 
J. Witte, eds.. Choice and Control in American Education: VoL 1. The Theory of Choice and Control in 
American Education (New York: Falmer, 1990), pp. 285-318. 

^ Disparate findings have emerged from these studies. For example, one analysis of the 
Milwaukee choice experiment found test score gains in reading and math, particularly after 
students had been enrolled for three or more years, while another study found gains only in 
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math, and a third found gains in neither subject. Jay P. Greene, Paul E. Peterson, and 
Jiangtao Du, “School Choice in Milwaukee: A Randomized Experiment,” in Paul E. 
Peterson and Bryan C. Hassel, eds.. Learning from School Choice (Washington, D.C.: Brookings, 
1998), pp. 335-56; Cecilia Rouse, “Private School Vouchers and Student Achievement: An 
Evaluadon of the Milwaukee Parental Choice Program,” Department of Economics, 
Princeton University, 1997; John F. Witte, “Achievement Effects of the Milwaukee Voucher 
Program,” paper presented at the 1997 annual meedng of the American Economics 
Associadon. On the Cleveland program, see Jay P. Greene, William G. Howell, and Paul E. 
Peterson, “Lessons from the Cleveland Scholarship Program,” in Paul E. Peterson and 
Bryan C. Hassel, eds., hxamingfrom School Choice (Washington, D.C.: Brookings, 1998), pp. 
357-92; Kim K. Metcalf, William J. Boone, Frances K. Stage, Todd L. Chilton, Patty Muller, 
and Polly Tait, “A Comparative Evaluadon of the Cleveland Scholarship and Tutoring Grant 
Program: Year One: 1996-97,” School of Educadon, Smith Research Center, Indiana 
University, March 1998. Greene, Peterson, and Du, 1998 report results from analyses of 
experimental data; the other studies are based upon analyses of non-experimental data. 

^ Results from the Dayton evaluadon after one year are reported in William G. Howell and 
Paul E. Peterson, ''School Choice in Dayton, Ohio: An Evaluadon After One Year," 
occasional paper (Harvard University, Program on Educadon Policy and Governance, 
February 2000). Second-year results for Dayton are described in Mardn R. West, Paul E. 
Peterson, and David E. Campbell, “School Choice in Dayton, Ohio After Two Years,” 
occasional paper (Harvard University, Program on Educadon Policy and Governance, 
August 2001). First-year results for Washington are reported in Patrick J. Wolf, William G. 
Howell and Paul E. Peterson, " School Choice in Washington, DC: An Evaluadon after One 
Year," occasional paper Harvard University, Program on Educadon Policy and Governance, 
February 2000). Second-year results for Washington are reported in Patrick J. Wolf, Paul E. 
Peterson, and Mardn R. West, “Results of a School Voucher Experiment: The Case of 
Washington, D.C. After Two Years,” occasional paper (Harvard University, Program on 
Educadon Policy and Governance, August 2001). First-year results from the New York City 
evaluadon are reported in Paul E. Peterson, David E. Myers, William G. Howell, and Daniel 
P. Mayer, “The Effects of School Choice in New York City,” in Susan B. Mayer and Paul E. 
Peterson, eds., Naming and Learning: How Schools Matter (^ 2 .shington, D.C.: Brookings, 1999), 
Ch. 12. Second-year results from New York City are described in David Myers and others, 
“School Choice in New York City After Two Years: An Evaluadon of the School Choice 
Scholarships Program,” occasional paper (Harvard University, Program on Educadon Policy 
and Governance, September 2000). All of the occasional papers mendoned in this note are 
available at the website address: http: / / data.fas.harvard.edu/ pepg/ . 

^ Baseline data from the D.C. and Dayton evaluadons are reported in Paul E. Peterson, Jay 
P. Greene, William G. Howell and William McCready, "Inidal Findings from an Evaluadon 
of School Choice Programs in Dayton, Ohio and Washington, D.C." occasional paper 
(Harvard University, Program on Educadon Policy and Governance, October 1998). 

Baseline data for New York City are reported in Paul E. Peterson, David Myers, Josh 
Haimson, and William G. Howell, “Inidal Findings from the Evaluadon of the New York 
School Choice Scholarships Program,” occasional paper (Harvard University, Program on 
Educadon Policy and Governance, November 1997. Both of these reports are available at 
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http:/ /data.fas.harvard.cdu/pepg/ . 

^ For a description of the kinds of private schools voucher students attended, see William 
Howell and Paul Peterson, with Patrick Wolf and David Campbell, The Education Gap: 
Vouchers and Urban Schools. (2002, Brookings Institution Press, pp. 36-37). 

’ The assessment used in this study is Form M of the Iowa Tests of Basic Skills, Copyright c 
1996 by The University of Iowa, published by The Riverside Publishing Company, 425 
Spring Lake Drive, Itasca, Illinois 60143-2079. All rights reserved. 

^Peterson et al, 1999; Myers et al, 2000; Howell and Peterson, 2000; West, Peterson, and 
Campbell, 2001;Wolf, Howell and Peterson, 2000; Wolf, Peterson and West, 2001; Howell 
and Peterson with Wolf and Campbell, 2002. All Reports available at 
http:/ /data, fas.harvard.edu/pepg/ 

^ Exact procedures for the formation of the control group are described in Jennifer Hill, 
Donald B. Rubin and Neal Thomas, “The Design of the New York School Choice 
Scholarship Program Evaluation.” Paper presented before the American Political Science 
Association annual meeting in Boston, MA, August 31, 1998. 

For a more extended discussion on these matters, see the initial reports for each city cited 
in footnotes 4 and 5. 

The Appendix to this report compares the characteristics of participants and non- 
participants in the second-year follow-up sessions. For a discussion of the weighting 
procedures used in these evaluations, see Howell and Peterson, 2000; Wolf, Howell and 
Peterson, 2000; Myers, Peterson, Mayer, Chou, and Howell, 2000. 

Difficulties were encountered in the administration of the first-year follow-up test at the 
initial pilot session in Washington, D.C. Test booklets were not available at the testing site 
for scholarship students in grades 3-8. Copies of the test arrived eventually, but the amount 
of time available for testing may have been foreshortened. Significant effects on reading 
scores are not apparent, but significant effects on math performance are evident, probably 
because the math test was the last to be administered. Statistical adjustments in the test 
score analysis take into account the special circumstances of the pilot session. 

We are indebted to Derek Neal for calling this interesting contingency to our attention. 

This procedure is discussed in Joshua D. Angrist, Guido W. Imbens, and Donald B. 

Rubin, "Identification of Causal Effects using Instrumental Variables," Journal of the American 
Statistical Association., 91 (1996), 444-462. The procedure, widely used by statisticians to 
correct for selection effects, was used to estimate the effects of actual class size reduction in 
Tennessee. See Alan Krueger, “Experimental Estimates of Education Production 
TnncxxonsT Quarterly Journal of Economics., 114 (1999), 497-533. 

Derek Neal, “The Effects of Catholic Secondary Schooling on Educational Achievement,” 
Journal of Labor Economics (1997) 15(1): 98-123. 




45 



45 



William N. Evans and Robert M. Schwab, ’’Who Benefits from Private Education? 
Evidence from Quantile Regressions," (Department of Economics, University of Maryland, 
1993); David Figlio and Joe Stone, "School Choice and Student Performance: Are Private 
Schools Really Better?" (University of Wisconsin Institute for Research on Poverty, 1997). 

John F. Witte, "School Choice and Student Performance," in Helen F. Ladd, ed., Holding 
Schools A ccountahle: Performance-Based Reform in E ducation (W as hington , D . C . : B roo kings , 1996), 
p. 167. 

Major studies finding positive educational benefits from attending private schools include 
James S. Coleman, Thomas Hoffer, and Sally Kilgore, High School Achievement (New York: 
Basic Books, 1982); John E. Chubb and Terry M. Moe, Politics, Markets, and America's Schools 
(Washington: Brookings 1990); Derek Neal, ‘'The Effects of Catholic Secondary Schooling 
on Educational Achievement,” of Eahor Economics (1997) 15(1): 98-123. Critiques of 
these studies have been prepared by Arthur S. Goldberger and Glen G. Cain, “The Causal 
Analysis of Cognitive Outcomes in the Coleman, Hoffer, and Kilgore Report,” Sociology of 
Education^ vol. 55 (April-July 1982), pp. 103-22; Douglas J. Wilms, “Catholic School Effects 
on Academic Achievement: New Evidence from the High School and Beyond Follow-up 
Study,” Sociology of Education^ 58 (1985), 98-114. 

Results from these evaluations are reported in Paul E. Peterson and Bryan C. Hassel, eds., 
Eeamingfrom School Choice (Brookings, 1998). 

This procedure was also employed in Krueger 1999. 

For a fuller discussion, see HoweU, Peterson, Wolf and Campbell, 2002, chapters 2, 6. 

^ Christopher Jencks and Meridith Phillips, eds.. The Black-White Test Score Gap (Washington, 
D.C.: Brookings, 1999). 

^ Also, see Ann Flanagan, Jennifer Kawata and Stephanie Williamson. 2000. Improving Student 
Achievement: WhatNAEP Test Scores Tell Us (Santa Monica, CA: RAND Corporation, 2000), 
p. 59. 

Duncan D. Chaplin. 1998. “Raising Standards: The Effects of High School Math and 
Science Courses on Future Earnings.” Virginia Journal of Social Policy and Law, 6:1, pp. 
111-126. 

A careful analysis of this question would require a fuUer examination of the probable 
economic benefits of test-score gains, the cost of private schooling, and the interest rates 
faced by various classes of potential borrowers. 

^ Since information about the distribution of students among schools is available from the 
first year of the Dayton evaluation, we were able to estimate the extent to which African 
Americans and non-African Americans were subject to uniform treatment simply by dividing 
the number of students in a category by the number of schools they attended. On the 




46 



whole, we found fairly low uniformity of treatment and not much difference between racial 
groups. For both African American and non-African American students receiving 
treatment, the degree of concentration among schools was, on average, just 3 students. 
Among students in the control group, the degree of concentration was 3 students per school 
for African Americans, 1.3 students for non-African Americans. According to this 
estimation, then, some difference in the degree of concentration between African American 
and non-African American students is evident but the difference is not large. One may also 
estimate the degree of uniformity of treatment by examining the percentage of students in 
the three schools serving the largest number of students. When estimated in this way, one 
again finds some difference between treatment and control groups. But in this case it is the 
non-African Americans who appear the most concentrated. A total of 37 percent of the 
African American treatment students enrolled in just three schools, as compared to 15 
percent of the African American members of the control group. For non-African 
Americans, these figures were 56 percent and 16 percent, respectively. According to this 
estimate, the non-black members of the Dayton experiment experienced a more uniform a 
dose of treatment as did the black students in the study. In sum, differences in uniformity of 
treatment are an unlikely explanation for the differential racial effects that have been 
estimated. 

A summary of findings from earlier studies is available in Paul E. Peterson, “School 
Choice: A Report Card,” in Peterson and Hassel, Learning from School Choice^ p. 18. Mark 
Schneider, Paul Teske, Melissa MarschaU, and Christine Roch, “Tiebout, School Choice, 
Allocative and Productive Efficiency,” paper prepared for annual meetings of the American 
Political Science Association, 1998, finds higher levels of parental satisfaction within New 
York City public schools, when parents are given a choice of school. 

Procedures for constructing the index as well as additional information on satisfaction 
levels are reported in Howell, Peterson, Wolf and Campbell, 2002, Ch. 7. 

Andrew J. Coulson, Market Education: The Unknoim History (New Brunswick: Social 
Philosophy and Policy Center and Transaction Publishers, 1999), p. 277. 

^ Estimates are based on information about Catholic schools in Manhattan, the Bronx, and 
Brooklyn from an unpublished memorandum submitted to PEPG from the New York 
archdiocese in August 1999. Public-school expenditure by school for the City of New York 
is available on the Board of Education website. 

Private school tuition rates were estimated in part from information provided in Lois H. 
Coerper and Shirley W. Mersereau. Independent School Guide for Washington^ D.C. and Surrounding 
jArea. IP^ ed. (Chevy Chase, MD: Independent School Guides, 1998). For schools not listed 
in this volume, information was obtained in telephone conversations with school staff. 

Some schools have a range of tuition charges, depending on the number of students from 
the family attending the school and other factors. The tuition used for this calculation is the 
maximum charged by the school. The tuition also includes aU fees, except for the 
registration fee, which is ordinarily treated as partial payment toward tuition. Figures are 
weighted proportionate to the number of students in the evaluation attending a particular 
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school. Public-school expenditure includes the costs of transportation and special education, 
which may not be provided by private schools. 

Data taken from the U. S. Department of Education, Office of Educational Research and 
Improvement. National Center for Education Statistics, Common Core of Data, School Years 
1993-94 through 1997-98. (Washington, D.C.: 2000). 

Provided with large differences in school size and other characteristics of schools for 
members of the treatment and control groups, we plan in the future reports to assess the 
extent to which the differences may indicate impacts of vouchers on other outcomes. 

Peterson, Myers, and Howell, 1998, Table 5. 

The reduction in class size in the Tennessee experiment was an average of approximately 
seven to eight students. Krueger, 1999. For a father discussion of this point, see Howell and 
Peterson, with Wolf and Campbell, 2002, pp. 158-64. 

John E. Chubb and Terry M. Moe, Politics, Markets and Yimenca*s Schools (Washington, D.C.: 
Brookings Institution Press, 1990). 

Thomas Hoffer, Andrew Greeley, and James Coleman, 1985. "Achievement Growth in 
Public and Catholic Schools," Sociology of Education, April, 58: 74-97. 

For very similar first-year results, see Peterson, Myers, and Howell, 1998, table 9. 

These response rates are similar to those in other randomized field trials that follow 
students over time. In his reanalysis of data from the Tennessee class-size study, for 
example, Krueger, while not providing annual attrition rates, reports that "only half the 
students who entered the project in kindergarten were present for all grades K-3." Krueger, 
1999, p. 506. 
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